Practical Collapsed Stochastic Variational Inference for the HDP

Recent advances have made it feasible to apply the stochastic variational paradigm to a collapsed representation of latent Dirichlet allocation (LDA). While the stochastic variational paradigm has successfully been applied to an uncollapsed representation of the hierarchical Dirichlet process (HDP), no attempts to apply this type of inference in a collapsed setting of non-parametric topic modeling have been put forward so far. In this paper we explore such a collapsed stochastic variational Bayes inference for the HDP. The proposed online algorithm is easy to implement and accounts for the inference of hyper-parameters. First experiments show a promising improvement in predictive performance. 1 Background We begin by considering a model where each document d is a mixture θd of K discrete topicdistributions φk over a vocabulary of V terms. Let zdi ∈ {1, ..,K} denote the topic of the i word wdi ∈ {1, .., V } in document d ∈ {1, .., D} and place Dirichlet priors on the parameters θd, φk. We have zdi | θd ∼ Discrete(θd) , θd ∼ Dirichlet(απ) , wdi | zdi, {φk} ∼ Discrete(φzdi) , φk ∼ Dirichlet(β) , where π is the top-level distribution over topics, and α and β are concentration parameters. While the dimensionality of K is fixed in latent Dirichlet allocation (LDA), we want the model to determine the number of topics needed. Consequently we follow the assumptions made by the hierarchical Dirichlet process (HDP) [1] of a countable but infinite number of topics, of which only a finite number is used in the posterior. Our prior π is constructed by a truncated sick-breaking process [2],